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ABSTRACT 

i 

The conversion of continuously varying data into a form 
acceptable to a digital telemetry system introduces error 
into the data. This paper examines the errors in this sig- 
nal conditioning process and derives expressions for them 
by using statistical methods. The standard calibration pro- 
cedure for analog to digital encoders is examined and found 
to be inadequate. A method of calibration is presented 
which overcomes the inadequacies of existing procedures. 
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THE ANALYSIS AND CALIBRATION OF 
ANALOG TO DIGITAL ENCODERS' 


by 

J. A. Loger 

Goddard Space Flight Center 


INTRODUCTION 

The primary data channel of a satellite is usually some form of digital telemeter. The infor- 
mation presented to this channel generally originates either from discrete state input devices such 
as counters, relays, or digital logic, or from continuous transducers such as voltage dividers, 
current probes and temperature sensors. In the former, signal conditioning is an error-free 
process of matching the discrete states of the input to the discrete states of the data channel. In 
the latter case, signal conditioning introduces error into the data since it must convert the con- 
tinuously varying data into discrete states (quantizing). Since there are usually several transduc- 
ers, the signal conditioner must also sample and multiplex inputs from each into the one data 
channel. 

This paper is concerned with the errors generated by the quantizing process. It attempts an 
analysis of the error contained in the theoretical model of the quantizer, and of the determination 
of error through the calibration of the actual device. Some space will also be devoted to the er- 
ror introduced by reconversion of digitized data, at the receiving end of the channel, into their 
original form. Errors due to sampling and multiplexing of data are not considered in this paper 
since most of the quantities being sampled change very slowly compared to the sampling interval; 
either their harmonic content is not of much interest, or the variation can be determined by sim- 
ple interpolation between data samples. For example, spacecraft battery temperature varies in 
terms of degrees per hour; with a sampling rate of 25 cps, the data appear virtually constant be- 
tween samples in time. 

Although the statistical equivalent of quantizing (rounding or grouping) was first examined by 
Sheppard (Reference 1) in 1898, the theoretical analysis of quantizers (analog to digital encoders) 
is more recent in origin. In 1948, Bennett (Reference 2) analyzed the power spectrum of the 
quantizing error; in 1956, Widrow (Reference 3) used mathematical statistics to examine an en- 
coder model. 


‘Also submitted to the University of Maryland in partial fulfillment of the requirements for the degree of Master of Science. 
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The calibration of analog to digital encoders has not been covered in any detail in existing 
literature. Whatever methods are mentioned generally do not provide adequate measures of en- 
coder errors, nor do they always lead to an 
accurate estimate of the encoder transfer func- 
tion. This paper is a partial attempt to rectify 
this situation. 


A typical spacecraft analog data signal con- 
ditioning subsystem is shown in Figure 1. Its 
output is one of several inputs to the main sat- 
ellite telemetry system. The transducer is an 
analog device which converts a physical quantity 
into a voltage. It can be represented by a two- 
port network whose transfer function is mono- 
tonic and continuous over its range of operation. 
Transducer calibration usually consists of finding a least squares fit for a set of measured input- 
output data point pairs. This procedure is well covered in the literature of statistics (References 
4, 5, and 6). The commutator is basically a multiple position electronic switch which selects dif- 
ferent transducer outputs for input to the encoder. In this paper, the commutator is assumed to 
contribute no error to the signal. 
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Figure 1— A transducer -encoder subsystem. 


ANALYSIS OF THE ENCODER 

The encoder can be represented by a two-part network with a transfer characteristic which is 
a monotonic step function with a range of 2" - 1 steps, where n is the number of binary bits in the 
encoder output. The general relation between the input and output is 

Q = kq, (k - q < H(v) < (k + j) q, k = 0, 1. 2 2" - 1, (1) 


where: 

Q = binary output 
q = quantizing interval 
v = input in volts 

H(v) = a continuous function of the input. 

The theoretical quantizer transfer function can be represented as: 

Q = kq, (k - q < x < fk+^Jq, k = 0, ±1, ±2, ... , (2) 
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where x is a continuous variable. Average quantizer gain is unity, whereas encoder gain is rep- 
resented by a function of the input, H(v), ideally of the form H = mv where m is a constant. The 


encoder model can therefore be represented by a 
quantizer proceeded by an "amplifier" of gain 
H(v) and x = H(v) . The term "amplifier" is used 
for convenience in the analysis even though the 
input voltage is converted to a continuous number 
rather than a voltage. Figure 2 is a block dia- 
gram of the analog to digital encoder model. The 
amplifier transfer function is shown in Figure 3a; 
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Figure 2— Model of analog to digital encoder. 
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Figure 3— Transfer functions. 


the standard quantizer transfer characteristic is shown in Figure 3b. In essence, the quantizer 
produces an output which is the integer nearest to the value of the output. This process introduces 
a quantizing error into the data. This error is not of a random nature; for any given input, it is 
known exactly. In application, however, the quantizer input is not available to the system which 
receives the quantizer output and therefore the quantizing error cannot be evaluated for a given 
datum. Since the input is not available but is known to vary, the associated quantizing error over 
a large amount of data can be assumed to be random. This allows the use of statistical analysis 
as a means of evaluating the average effects of quantizer error on the quantizer output. 

In a statistical analysis, the items of interest are usually the mean of a random variable 
given by 


u 


x 



x f (x) dx 


( 3 ) 
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where 


u = mean of x, 

X 7 

f ( x ) = probability density function of x(p.d.f.), 
and the variance; 


a 


2 

x 


[ ( X “ U x ) 2 f ( X ) dx • 


(4) 


The square root of the variance is the standard deviation cr x . The mean u x is the average value of 
the random variable x. In the case of error, it is desirable that the average value be zero. The 
standard deviation cr x is the rms deviation of the random variable from its mean. It is a measure 
of the scattering of the data. For example, if the mean of the error is zero, then is the rms 
error. In general, 


M n 



( x ) n f ( X ) dx 


(5) 


where M n is the nth moment of the random variable x. The characteristic function (c.f.) of a proba- 
bility density function is defined as 


4> x (t) = e itx f (x) dx , (6) 

•'-00 

it has the property that 

d"0 x (t) 


where i - and the nth moment of f(x) exists. This property will be used later to find the 
variance of a quantized variable. Equation 7 gives the moments around zero. cr x 2 is a moment 
around u x . The two moments are related by 


(7) 

t=o 


a 


2 

x 


m 2 - 


( 8 ) 


Also 


<*>„ (0) = 1 • 


(9) 
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This can be seen from Equation 6 and 


* / -a 


f ( x ) dx - 1 . 


( 10 ) 


The c.f. also has the property that if x and y are independent random variables, then: 


^(x + y) (t) = ^,(t)0 y (t) . 


(U) 


If the input to the quantizer is a random variable, then the error produced by the quantizer 
must in some way alter its p.d.f. Examination of the input and output p.d.f. f s of the quantizer 
should reveal the effect. 

Since any value of x in the range kq ± q/2 is read as kq , the probability that the output of the 
quantizer is kq must be: 


g(kq) 


/•kq+q/2 

■'kq-q/2 


f ( x ) dx 


(12) 


where f(x) is the p.d.f. of the random variable x . The quantizer converts a continuous p.d. f . into 
a series of discrete probability pulses as shown in Figure 4. The function g(kq) can be viewed as 



Figure 4— Functions relating to the quantizer. 
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a function g(y) which has been sampled at y = kq, k = ±1, ±2 . . . . Now 


r y + q/2 

g(y) = f(x) dx 

•/y-q/2 


I f(x 

•'-00 


) U(y - x) dx , 


( 13 ) 


where 


therefore 


U(y - x) = 1 for y - q/2 < x < y + q/2 
= 0 otherwise; 


g*( y ) * ^ e(y ^ 5 < y_kq) 


represents the sampled function and S(y) is the Dirac-delta function. The sampled function g*(y) 
is the p.d.f. of the output if f(x) is the p.d.f. of the input. 

In its present form, g*(y) is too awkward to provide much information; however, the charac- 
teristic function of g*(y) will provide more information. This is true partly because of Equation 7, 
which states that if the moments of the p.d.f. of a random variable exist, they may be found by the 
various derivatives of the c.f. at the origin. Equation 13 is a case of the convolution integral 

/»00 pCD 

h(y) = I r(x) s(y - x) dx = I s(x)r(y-x)dx . (14) 


The important property of this integral is that the Fourier transform^ (t) of h(y) is 


' ‘M* )</>.(*) 


(15) 


6 



•The c.f . of g*(y) is 


,<o . r 

•'-a 


</>„. (t) - I e ity g*(y)dy 


/.CD 00 

■ L- L 

k=-a> 


g(y) 8(y-kq) 


dy 


eitkq g(kq) • 

k=-oo 


(16) 


This relation is a Fourier series representation whose fundamental period is 2rr/q . Evidently, the 
characteristic function of the quantizer output is periodic with period 2n/q. The waveform of 
4> g * (t) can now be found with the aid of Equation 15. 


It can be shown (Reference 9) that the coefficients g(kq) of the Fourier series are equal to the 
values of the Fourier integral of the typical waveform of <£ g * (t) evaluated at kq, k = ±1, ±2, ... , 
times the fundamental frequency. Thus 

<*> = \ ^ ■ ^r) ; 

k=-oo 

but 



1. 

q 



f (x) U(y -X) dx 


and therefore, by Equations 6 and 15, 


(O “ 



(17) 


(18) 


This relation does not appear to be better for analysis purposes than that shown in Equation 16. 
However, if a restriction is placed upon <f> { (t), it does lead to useful results. Equation 7 shows 
that the p.d.f. moments are determined by differentiating the c.f. at the origin. If the quantizing 
width q is made fine enough so that the c.f. of the input (the typical waveform) is zero for 1 1 | > 2 n/q , 
then there would be no contribution to the c.f. at the origin from other parts of the c.f. Consequently, 
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the derivatives of the c.f. at the origin would be the derivatives of 


sin 

= ^f(o n " • < 19 ) 

2 

By Equation 11, this is the characteristic function of the sum of two independent random var- 
iables, one of which is the input and the other is a uniform distribution of width q and uniform 
height l/q . This can be condensed into the following statement: If the characteristic function of a 
random input is zero for |t| > 27 r/q, then the error due to quantization can be considered as an in- 
dependent random variable with the uniform distribution described above, and the output is the 
sum of the two independent random variables, the quantizer input x, and quantization noise N. 

The restriction implies that the p.d.f. of the input tapers off smoothly to zero at the ends of 
its range, since the Fourier transform of such a function has a definite "bandwidth." Taking the 
derivative of Equation 19 and letting 

s in ^ 

‘M*) = — u[ — ' 

2 

we have 

(t) d*> x (t) (t)l 

dt - dt [ 4 >n (*>] + (O [ dt J • 



By Equation 7, setting t = o, 


U Q - U * 


since <£ N (t) = l at t = 0 and d^ N (t)/dt = o at t = o . Since x and N are independent, 


‘ ^ + <yj- 


Using Equation 4, 


r r q/2 i 

N 2 f(N)dN = -N 

J-oo J-q/2 


dN 


N 3 

3 q 


q/2 

-q/2 


Q 

12 


Thus 


a 2 + % and a 2 

X 12 X 


a Q ‘ 12 


( 20 ) 


( 21 ) 


( 22 ) 
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Equations 20 and 22 are the first two of a series of correction formulas first derived by 
Sheppard (Reference 1) to find the various moments of the input distribution from the moments of 
the output of a quantizer. Widrow (Reference 3) analyzed the behavior of a quantizer with a Gaus- 
sian input. The results showed that for quantization intervals as large as twice the standard devia- 
tion of the input, the error in the calculated variance of the input, using Equation 22 is 3 percent. 
This error increased drastically for higher moments. The probability that a Gaussian variable 
deviates from its mean by more than 3a is about .0027 ; 99.7 percent of the fluctuations are less 
than 3 cr in magnitude; and 99 percent of the fluctuations are less than 2.6a. This implies that, for 
an approximately Gaussian variable, useful estimates of mean square error can be made even if 

the range of the variations is only about 2.5 quantization levels. The errors in the corrections for 

2 2 

Gaussian input are due to the fact that the c.f. of a Gaussian variable decreases in size as a 
and therefore never attains zero. It is used as an example to provide a feel for the size of quan- 
tization interval required for useful results. 

In general, the narrower the range of the random variable, the wider the characteristic func- 
tion. Consequently, a rule of thumb for the size of the quantization interval could be based upon 
the range of the random variable. If the random fluctuations in the input are roughly Gaussian, 
then a quantization width of one -third of the range is adequate. This leads to an error of only 3 
percent in the estimation of the variance of the input. If the distribution of the input is known, a 
quantization width can be determined more accurately by examination of its characteristic func- 
tion. This process is an attempt to insure the independence of the quantization noise and so make 
the error analysis of an encoder relatively simple. If such independence cannot be assumed, the 
error analysis may require more sophisticated and time-consuming techniques. 

If the input to the quantizer consists of a sum of independent random variables of which only 
one satisfies the above criterion, then the sum satisfies the criterion. If the input x = y + z + a , 
where y, z and a are independent random variables, then by Equation 11: 

mo = [mo] [<m o] [mo] • 

If y satisfied the criterion, then <f> (t) = 0 for [ 1 1 >2rr/<\. Consequently, </> x ( t) - 0 for |t| >Vq. 
Because of this property, it may be useful purposely to add noise of known amplitude to the analog 
data to aid in recovering the moments of the data. If, for example, the range of the input random 
variable were only one quantization interval, the input would not satisfy the conditions for the in- 
dependence of quantization noise. If noise of rms amplitude q/2 were added to the input, the con- 
dition would be met and the variance of the input could be calculated. This technique will be il- 
lustrated in the section entitled "The Calibration Procedure. 11 

Finally, if the distribution of the input to a quantizer is known, uniform quantization may not 
necessarily be the optimum form. One investigation of this topic (Reference 7) has shown that, 
for minimum quantization noise, some form of nonuniform quantization may be required. However, 
if the distribution is not known, the encoder system could consist of some form of adaptive quan- 
tizer which examines the input data and adjusts the quantization intervals accordingly. 
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THE CALIBRATION PROCEDURE 


The results of the previous section were based on certain assumptions about the statistical 
properties of the quantizer. These properties are in turn based upon the average effects of a 
quantizer upon a large number of input data points. The theoretical quantizer is not a truly ran- 
dom device. For example, if the input to the quantizer is 4.3725 . . . then the output is 4.0000 . . . , 
always. This is in contrast to a truly random device such as an information channel whose output, 
for a given input symbol, has a finite probability of being any one of several symbols. During 
actual use, when only the output of the encoder is available, only statistical techniques are avail- 
able for estimating the input and expressing error iri the data. During calibration, however, this 
is not the case. 

Calibration is a process for determining the real gain function of a device as opposed to the 
theoretical. In the previous section, the theoretical encoder was represented as a quantizer pre- 
ceded by a linear amplifier. The real encoder can be represented by the same quantizer preceded 
by a nonlinear amplifier and perhaps a d.c. offset. The offset may be required to account for en- 
coders which produce an output other than the midpoint of the quantization interval. The model 

Since the quantizer is assumed to be ideal, 
its transfer characteristic is known. Conse- 
quently, the purpose of the calibration is to de- 
termine the combined transfer characteristic 
of the amplifier and offset. This is complicated 
by the fact that the output of the latter two de- 
vices is observable only through the masking 
effect of the quantizer. In addition to the quan- 
tization error, there are fluctuations present in 
the real encoder transfer function due to varia- 
tions in circuit parameters, changes in environment, etc. In the model shown in Figure 5, these 
can be viewed as variations in H(v) and the offset. As a result, the exact transfer function for the 
real encoder does not exist. There is, however, an average function around which the real one 
fluctuates. The purpose of calibration is to estimate this average curve. Calibration accuracy is 
measured by the closeness of the estimated average to the true one. The theory of estimation of 
functional relationships has been extensively treated in the literature of statistics (References 4, 

5, and 6). 

A method in general use for estimating the transfer function is the method of least squares. 
This technique minimizes the sum of the squares of the differences between the observed data and 
the estimated transfer function. In the case of the encoder, if Y = H(x) is the transfer function of 
the amplifier, then the least squares estimate H*(x) of H(x) is the one that minimizes 

Z] b ■ 

j = i 


for the actual encoder is shown in Figure 5. 



Figure 5— Encoder model. 
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where (y. , Xj j , j = l, 2, . . . m, are the observed output-input pairs. The method is based upon the 
Gauss-Markov theorem: 

If the model for the device is expressible in the form Y = W + E, where 



Vi 


X n X 12 ... Xjp 


*r 


e i 

Y = 


X = 

X 21 

e = 

*2 

E = 

e 2 




X ml X mP 


>- 


>- 


with the average error u e = 0, and if the errors e j are independent of the y. and have a common 
but unknown variance a e 2 , then the vector of coefficients 0 which minimizes (Y- \0) * (Y -xS\ where 
* indicates transpose, is the linear, minimum variance unbiased estimator of 0. The term "un- 
biased” implies that = 0 ; "minimum variance" means that the variance of 6 is less than or 
equal to the variance of any other estimator of 0 , and the term "linear" means that the estimator 
of 6 is expressible as a linear combination of the observed random variables Y. 

This theorem allows the use of variety of mathematical models to represent the transfer 
characteristics of an electronic device. The only restrictions are (1) that the model be linear in 
the coefficients, and (2) that the errors in the observed outputs be statistically independent. The 
first condition permits models of polynomial form 

p 

v = . 

k = l 

and transcendental models of the form Y = e x + 6> 2 e ±x and Y - 0 sin x, but not models of the form 
Y = e x +0 2 e^ 3 * . However, even the latter model can be transformed into an acceptable form. Re- 
striction (2) above is satisfied in most calibration problems if some care is exercised. Since 
an encoder is a complex device, a fluctuation in the output is the sum of many variations within 
the equipment. As long as one of these variations is not unduly large, the errors in a given set of 
outputs can be assumed to be independent. Variables of the encoder which do cause large errors 
in the output must be controlled. For example, temperature and burn-in time have a considerable 
effect upon the encoder. For proper calibration, measurements must be made at different tem- 
peratures and times and transfer functions must be determined for each set of conditions. 

The major drawback of the least squares method is that although given the model, it will de- 
termine the best estimates of the coefficients, it will not determine whether the model chosen is 
the best one for the data. Unfortunately, present methods of determining the best model are mostly 
intuitive. If polynomials are chosen, then the residual variance defined by 


s 


2 


1 

m -P 
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Y 
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L 

L- = 1 


"I 2 
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can be examined. A sufficiently large (~10) decrease in the residual variance due to using the 
next higher order polynomial can be taken as an indication that the higher order polynomial fits 
the data better. This is discussed in Reference 4. Even this technique can be misleading; for 
example, an (m- 1) order polynomial will fit m data points exactly but, since the data points were 
observed with error, there is no assurance that the resulting polynomial is a good approximation 
of the underlying model. 

The deficiencies in present encoder usage and calibration techniques stem either from mis- 
application of the least squares method, or from misunderstanding of the statistical properties of 
the encoder. The usual calibration process is fairly simple. An accurately known input voltage 
is stepped in equal increments through the range of interest, and for each input voltage an output 
binary number is recorded. The process is repeated several times; the outputs at each voltage 
are averaged over the several sweeps; and either a least squares fit is made of the inputs vs. 
averaged output pairs, or the averaged outputs are connected by straight line segments repre- 
senting the transfer function interpolated between measured values. If there is a fluctuation in the 
binary output for any given input, the binary number which occurs most frequently is recorded as 
the output for that particular sweep. The calibration procedure is repeated at various tempera- 
tures (for example, -40°C, 25°C, 60°C), and the group of calibration curves is usually decorated 
with the cryptic remark "±1 count" or ,T ±* counts." 

There are several reasons why this procedure is incorrect. First, the use of fixed input 
voltages may lead to a systematic error in calibration. In the model shown in Figure 5, the output 
corresponds to the midpoint of the rounding interval. There is, thus, a unique value of input which 
actually produces an amplifier output equal to this midpoint, but with no rounding error. The in- 
puts in the calibration procedure generally do not correspond to the midpoints of rounding inter- 
vals. Consequently, the corresponding amplifier outputs are rounded off. The fluctuations in the 
amplifier and offset will cause a variation in the amount of roundoff, but the average round-off 
will not be zero. For example, if, for a given input, say 3.6500, the amplifier output is 72.647 • • ■ , 
the output for the encoder is 73 and the rounding error is .353 • * - ; fluctuations in the encoder 
may cause the amplifier output to vary, but the average of the variations will still have a rounding 
error of .353 • • • . The least squares method assumes that the average error in an output due to 
any input is zero. Since this assumption is not true, the calibration procedure introduces a bias 
into the measurement by associating the wrong input with the mid-point of the rounding interval. 
This illustrates the misapplication of the statistical properties of quantization noise. 

Second, the use of fixed increment steps in the input violates the independence of errors con- 
dition of the Gauss-Markov theorem. The situation is illustrated in Figure 6; v x is applied to the 
input and Q x is observed as the encoder output. The amplifier output is Yj and the rounding error 
for the particular measurement is . The input is stepped by an amount Av to v 2 , and the ampli- 
fier output is now Y 2 ; but Y 2 - H( Vl + Av). If H(v) = mv + b and q = 1, then Y 2 - Yj = mAv, 
r 2 ” r j + (mAv - 1) and, in general, r n - T% + (n - 1) (mAv- 1). For the linear case, the rounding 
errors are a function of the output and therefore are not independent. This changes the values of 
the coefficients of the calibration curve. The same development could be used to show error 
dependence in higher order models. 
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Q + l 
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Figure 6—Use of fixed increment steps. 


Finally, the choice of the most frequently occurring output truncates the range of the output 
random variable; the averages calculated from such data will therefore not be unbiased. This 
leads to even further error in the calculated calibration curve. 

From the foregoing, it is apparent that the standard calibration scheme leads to input-output 
data pairs that do not provide a good estimate of the encoder transfer function. The calibration 
procedure is based on the assumption that, on the average, an output uniquely corresponds to an 
input. This is not so for the encoder. The corresponding assumption for use of the least squares 
method is that the underlying model is continuous and monotonic. The least squares method can- 
not be used for discrete and discontinuous models. The overall encoder transfer function clearly 
falls in this category. 

It was mentioned previously that if the encoder is represented by a quantizer preceded by an 
amplifier, the only unknown in the model is the amplifier transfer function. This transfer function 
is continuous and monotonic. The offset is usually a known quantity since it is a function of en- 
coder type and can be incorporated as a term in the amplifier transfer curve. The output of the 
encoder is the output of the amplifier rounded to the nearest integer. At a discrete set of inputs, 
the output of the encoder equals the output of the amplifier. Since the average effects of the fluc- 
tuations in the amplifier outputs are zero at these input points, the average error in the encoder 
output must be zero. If the input values were known, then the amplifier transfer curve would be 
known. In other words, at a given set of discrete input values, the transfer function of the encoder 
equals the transfer function of the amplifier. This obvious and very basic property of the encoder 
suggests a calibration method which avoids the pitfalls of the standard one. 

For a given input v . the output of the amplifier is a quantity which is rounded to Q J by the 
quantizer. Because of fluctuations within the amplifier, its output is occasionally rounded to some 
other, incorrect, number. The closer the amplifier’s average output is to the edge of the rounding 
interval, the more frequently this occurs; it depends upon the probability that the size of the ampli- 
fier’s output fluctuations exceeds the difference between the average output and the edge of the 
rounding interval. 

If the fluctuations can be assumed to occur in both positive and negative directions with equal 
probability, the point of minimal variation in encoder output is when the average amplifier output 
is equal to the midpoint of the rounding interval. Also, at the midpoint, encoder output variations 
are positive and negative with about the same frequency. Thus, if the input to the encoder is varied 
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over an interval corresponding to a variation of about one rounding interval of the amplifier out- 
put, then the point of minimum and bidirectional variation in encoder output should be the midpoint 
of the rounding interval. The input and encoder output are now a data pair which can be used in 
determining the amplifier transfer curve. Strictly speaking, the procedure is estimating the 
median of the amplifier output fluctuations, not the midpoint of the rounding interval. The median 
(u 5 ) is such that 


r» r 

f (x ) dx “ I f (x ) dx 

J— CD J U g 

and is the point around which the fluctuations are positive and negative with equal probability. This 
corresponds to the midpoint of the rounding interval if the amplifier output p.d.f. is symmetrical. 

If the fluctuations can be assumed to be due to a large number of minute variations within the 
encoder, then this assumption is valid. 

In actual measurement, because of the rounding effects of the quantizer, there will be some 
error in finding the coincidence of amplifier output and rounding interval midpoint. The input may 
correspond to an amplifier output which is not exactly equal to the midpoint. For small amplifier 
fluctuations there can be a small interval over which there is no appreciable fluctuation in the 
encoder output, and consequently the coincidence of amplifier output and rounding interval midpoint 
would be hard to identify. This could be avoided by arbitrarily adding a small amount of noise of 
zero average value to the input to increase the output fluctuations. In any case, (1) the error in- 
volved in the above procedure is much less than the size of the rounding interval; (2) on the average, 
the estimate of the midpoint should be correct; and (3) since the procedure is only concerned with 
estimating the midpoint of one rounding interval, the errors from one interval to the next are 
independent. Furthermore, by the formation of the model, the midpoints are parts of the continuous 
amplifier transfer curve. Any errors in determining the midpoint may be viewed as random errors 
around the midpoints. On the basis of the above, the data pairs satisfy the conditions for the 
application of the least squares method. 

The question of exactly how the two procedures differ might arise since they both take associ- 
ated input-output measurements of the encoder and fit them to a curve. Both use the discontinuous 
and discrete output of the encoder, yet one procedure permits the fitting of a continuous curve, and 
the other does not. The answer lies in the type of data being collected. The standard procedure 
does not recognize that the same output may be produced by a range of inputs, and consequently 
the data pairs obtained cannot determine a unique curve for the encoder. The other procedure 
attempts to find the input-output pairs for which the rounding error is zero; these are unique data 
pairs. Consequently these data pairs will determine a unique curve for the encoder. 

One quantity associated with the encoder has not yet been discussed; the size of the fluctua- 
tions of the amplifier. In the normal calibration of a continuous device, this would have been 
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estimated by the square root of the residual variance mentioned previously. Unfortunately, the 
residual variance obtained by fitting a least squares curve to the data obtained by the above men- 
tioned method, is not directly related to the size of the amplifier fluctuations. If the encoder input 
were set at one of the points determined to correspond to a rounding interval midpoint, the situa- 
tion would correspond to the one described in the encoder analysis. The amplifier output would 
fluctuate with some distribution about the midpoint of the rounding interval. If the fluctuations 
were large enough, then the rounding errors could be assumed to be independent of the errors due 
to the fluctuations. And the variance of the amplifier output would be 



Since it is not known whether the amplifier output satisfies the independence criterion, noise, 
which is known to meet that criterion, must be added to the input. As was mentioned earlier, if 
any one of several independent random variables meets the criterion, then the sum of the random 
variables also meets the criterion. It will be shown later that the variance of the noise in the 
amplifier output due to the noise in the input is given by 


dH( v) 


dv 

V . 

J 


where the derivative of the amplifier transfer curve is evaluated at the input value. Consequently, 
the variance of the output is 



and therefore 



(23) 


(24) 


The noise at the input can be obtained from a random noise generator and must have a zero aver- 
age value. The rms value of the noise is cr v . If the noise generator is roughly Gaussian, cr v must 
be just large enough to produce a a Y of q/2, since a larger value might obscure the internal fluc- 
tuations of the amplifier. A number of samples are taken with the noise applied and the variance 
cr £ 2 is estimated by the formula 


c 2 
b E 


£ 


(gi -Q) 2 

m - 1 
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where m is the number of samples, Q is the average of the outputs, and S E 2 is the estimator of the 
encoder variance. The number of samples required is not critical; about 30 should be sufficient. 
The number obtained by the use of Equation 24 should be a useful indicator of the mean square 
error in the encoder output due to the analog portions of the encoder. It would be useful to repeat 
this process at several points along the calibration curve and to take an average of the variance at 
these points. 

The calibration curve should be accompanied by the estimated variance, not by the presently 
used statement of the range of variation. This allows the experimenter to judge the effects of the 
encoder upon his data. The statement ±1 count does not provide information about the rms error 
in the encoder and, consequently, does not allow comparison of the encoder error with errors in 
other parts of the system. 

During use, the amplifier calibration curve can be used as the transfer function for the en- 
coder, and the effects of quantization can be considered as part of the total error of the subsystem. 


ERRORS IN THE TRANSDUCER-ENCODER SUBSYSTEM 

Several other sources of error exist in the transducer-encoder subsystem. Encoder error is 
only a part of the total error. For example, a transducer with an accuracy of ±1 percent full-scale 
is a good transducer. Suppose that the ±1 percent figure means that an error of 1 percent of the output 
represents a deviation of 3 from the average. Assuming that the encoder is perfect and only has a 
rounding error, the rms transducer error of 0.33 percent would produce an error component of 0.33 per 
cent of full-scale of the encoder output. If the encoder is an 8-bit encoder, full-scale is 255, and 
the quantizing mean squared error is [^(100/255) x >/ 1/12 ] 2 = (.013%) 2 of full-scale. Since the 
transducer satisfies the quantizing criterion, the two errors are independent and the variances can 
be added. Therefore, the mean squared error of the subsystem output is .111 + .013 = .124% 2 . 
Obviously, the error due to the transducer is a significant part of the total error. 

The above example uses percent error as a means to an end. The transfer of input error to 
the output of a device is not usually that simple. Only the assumption of a perfect encoder— i.e., 
amplifier function linear and invariant— allowed the use of percentages. In general, if the output 
of a device is related by a function, v F(x),to the input, then an error in an input x x of magnitude 
Ax produces a change in the output of e f ( x 1 Ax) -f(x 1 ). Now 

. F(x , + Ax ) ~ F (x,) 

Ax Ax 


and if Ax is small. 


! ( X > ' Ax )_~ F ( x i) dF(x) 

Ax dx 
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thus 


dF[x) 

dx 


Ax 


and 


dF(x) 


dx 

x i_ 


Ax 2 


By the definition of variance, 


dx 


°Ax and CT e 


dF(x) 


dx 

x i 


a kx 


(25) 


The difference F(x +Ax) -F(x) can be expanded as a Taylor series around Xj. If Ax is small and 
the higher derivatives of F(x) are small, then the remaining terms in the expansion are insignificant 
and the relations hold. 

Equation 25 can be used to express the error present in the output of an encoder due to the ef- 
fects of the transducer-encoder subsystem. For a given input to the transducer with output v } 
and a variance cr T 2 , the expected mean squared error in the output of an encoder with analog trans- 
fer curve H( v ) and variance cr A 2 is: 


a 


2 

E 


dH(v) 

dv V. 



(26) 


This expression shows that the error in the encoder output may partly depend on the magnitude of 
the input. If so, then errors of a certain percentage of full scale in the input will not produce the 
same percentage error in the output. For this reason, it is best to transpose all errors to the 
output before converting to percent full-scale. Equation 26 shows that the rounding error is only 
one part of the total error associated with the encoder output. 

The information contained in the output of an encoder is not generally useful in its binary form. 
Consequently, it is converted to the physical units of the input transducer. This requires the in- 
version of the encoder and the transducer transfer functions and produces the relations 
v } - H~ 1 (Qj) and x } = F " 1 (vj where v = F(x) is the transfer function of the transducer. By Equa- 
tion 25, the variance of the encoder input v } as calculated from the encoder output at Qj is: 



(27) 
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Adding the transducer variance and again using Equation 25 gives the variance associated with 
the value of the input to the subsystem as calculated from the encoder output: 


dF(x) 


dx 

X . 
1 


( 28 ) 


x j and v j are the estimated values of the input as determined from the calibration curves. 

The transfer function of the encoder will usually be linear enough, for the purposes of calcu- 
lating o- v 2 , so that dH(v)/dv can be assumed constant. This does not apply generally, to the trans- 

J 

ducer transfer curve and, as a result, error estimates should be made at several points. 

The quantities used in actual calculation are only estimates of the underlying model param- 
eters and therefore the results are only approximate. However, they do provide an estimate of the 
size of error in both the observed output data and the calculated input of the transducer- encoder 
subsystem. 


CONCLUSIONS 

If certain conditions are satisfied, the errors produced by the rounding in an analog-to-digital 
encoder may be assumed to be independent of the input. This assumption leads to simple expres- 
sions for the total error contained in the output of a transducer-encoder subsystem. The errors 
contained in the input value, as calculated from the observed output value, depend upon the trans- 
fer characteristics of both the encoder and the transducer. 

The encoder transfer curve cannot be determined accurately with standard calibration tech- 
niques. Such techniques lead to errors in the calibration data which violate the assumptions re- 
quired for regular curve fitting methods. In this paper a calibration procedure has been derived 
which overcomes these difficulties and permits the use of regular curve fitting methods. 
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